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METHOD AND APPARATUS FOR COLLABORATIVE 

ANNOTATION OF A DOCUMENT 

BACKGROUND TO THE INVENTION 

5 1. Field Of The Invention 

This invention relates to the field of facilitating collaborative annotation of 
a document. The invention is especially suitable for (but not limited to) 
permitting collaborative annotation of a document by users located at different 
geographical locations. The invention is also especially suitable for (but not 
10 limited to) use with a real-time conferencing system. 

2. Description of Related Art 

The main problem addressed by a collaborative annotation system is 
how to enable users located at different locations, for example different 
geographic sites, to annotate a common document, and to view each other's 
15 annotations. 

This is especially desirable in a real-time environment, such that the 
users can work on the document together (for example in combination with a 
conferencing system, such as a video conferencing system). 

U.S. Patent No. 5,826,025 describes a system which supports 
20 collaborative annotations on Web pages. However, the system only permits 
annotations in the form of character sequences obtainable by a standard 
computer keyboard. In other words, each user has to enter the annotations 
using a keyboard. 

U.S. Patent No. 5,781 ,727 describes a system which permits annotations 
25 made by users at remote computers to appear on the displays of other 
participants. The system can be added to existing document sharing systems 
such as those described in U.S. Patent Nos.: 5,220,657; 5,583,993; and 
5,781 ,732. However, the annotations are restricted only to standard shapes, 
symbols and diagrams similar to those found in a graphics "paint" program. 
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U.S. Patent No. 5,897,648 describes a collaborative annotation system 
which supports handwritten annotations. However, the system requires a 
special digitizer pen and an x-y digitizer base on which the paper document 
must be placed so that the annotations can be recorded and digitized. Such 
5 apparatus is expensive, and requires considerable desk space making it 
inconvenient to use. Additionally, all handwritten annotations have to be 
inputted using the digitizer, and so the system cannot deal with any 
documents which have already been annotated. 

SUMMARY OF THE INVENTION 

10 Broadly speaking, in contrast to the prior art, the invention provides a 

collaborative annotation technique in which users can make handwritten 
annotations to original paper documents. The invention provides a system in 
which each operator uses an image capture device is to capture a digital 
image of the document including any handwritten annotations, and the image 

15 is processed to extract images corresponding to the annotations. These 
extracted images are distributed to other users for display. Each user may 
therefore be able to view a combined image of the document on which all of 
the individual annotations are overlaid. 

The invention thus improves on the prior art in several significant 
20 respects. 

Firstly, the annotations are not limited to only special shapes, or 
keyboard entries. Almost any annotation can be accommodated in 
handwritten form. 

Secondly, the system does not require an operator to use an expensive 
25 or space-consuming digitizer x-y device and pen to enter the annotations. 
Instead, the annotations are identified and extracted by processing a captured 
image of a document on which the annotations have been made. 

Thirdly, each operator can make the annotations directly on a paper copy 
of the document. This is by far the most convenient technique for a user, 
30 because users still prefer to handle, and to make annotations on, a paper 
document rather than on an electronic representation of the document. 
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Additionally, when authors meet to discuss a document they are writing, 
they frequently work with paper drafts - annotating the parts of the document 
they plan to amend. The present invention can enable users to annotate 
documents in the way in which they are already most familiar, and 
5 automatically display the combined annotations from a plurality of users. 

Preferably, each identified annotation is processed and stored as a data 
object. By separating the annotations in objects, the annotations can be 
independently manipulated. 

Preferably, the system is able to update a user's display with any new 
10 annotations made by another user. 

BRIEF DESCRIPTION OF THE DRAWINGS 

A non-limiting embodiment of the invention will now be described, by way 
of example only, with reference to the accompanying drawings, in which: 

Fig. 1 is a schematic block diagram showing a collaborative annotation 
15 system; 

Fig. 2 is a schematic flow diagram illustrating operation of the system of 
Fig. 1 ; 

Fig. 3 is a schematic illustration of a data format representing an 
annotation object; 

20 Figs. 4(a), (b) and (c) are schematic representations of annotations 

made to paper document by different users; and 

Fig. 5 is a combined view showing the combination of the annotations of 
Figs.4(a), (b) and (c); 

DETAILED DESCRIPTION 

25 Referring to Fig. 1, a collaborative annotation system 10 comprises a 

network server 12 communicating with a plurality of workstations 14. In this 
embodiment, four workstations 14 illustrated. However the number of 
workstations may be less, or it may be increased, to accommodate any 
number of users as desired. 
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The network server 12 may communicate with the workstations 14 by 
any suitable communications medium, including a telephone or other dial-up 
network, or a local area network, or a wireless network. The communication 
medium may include the Internet, or a video conferencing channel. 

5 The relative locations of the workstations 14 is not important, as the 

workstations operate independently of each other. In particular, this 
embodiment enables the workstations to be at completely different 
geographical locations. 

Each workstation comprises a computer processor 16, a display unit 18 
10 and a document-imaging device 20. It is preferred that the imaging device 20 
be a camera which can focus on a document 22 being worked on. However, 
any suitable imaging device, such as a scanner may be used; the user would 
then have to re-scan the document whenever he wished to update the image 
of the document. 

15 In contrast to the prior art, one of the principles of this embodiment is that 

users can make handwritten annotations to a paper document, and the 
handwritten annotations from each user are extracted and combined with 
other user's annotations for distribution to each user. This provides the 
advantage that a user is able to work directly on the paper document, and the 

20 user can make any type of annotation he desires, as he is not limited only to 
specific electronic annotations. 

Fig. 2 illustrates an overview of the system-wide operating method. 

The system enables users to make handwritten annotations to a 
document prior to use with the collaborative annotations system. At a first 

25 step 30, each user operates the imaging device 20 to capture an image of the 
document 22, so that any existing annotations may be extracted. The 
extraction may either be performed locally at each workstation 14, or centrally 
on the server 1 2. Local identification of the annotations has the advantage 
that the processing may be performed "offline" before the workstation is 

30 coupled to the server 1 2 for a collaborative session. 

The initial annotations may be identified by comparing each image with 
an original "clean" image without the annotations. However, this technique 
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relies on having an accurate "clean" image for each user, which may prove 
difficult if each user has received the document electronically and might have 
printed it differently. 

An alternative, more elaborate, technique is to use an algorithm for 
5 discriminating machine printed symbols from handwritten annotations on the 
basis of the image characteristics. Such a technique is described, for 
example, in U.S. Patent No. 5,384,863, the teaching of which is incorporated 
herein by reference. 

Each annotation is stored in the form of an "object", the format of which 
10 is illustrated in Fig. 3. Each object comprises: a document identifier 32 for 
identifying the document on which the annotation was made; a location 
identifier 34 for identifying the location of the annotation in the document; a 
time stamp 36 indicating the time and/or date the annotation was detected; an 
author identifier 38 for identifying the author (or at least the originating 
15 workstation) of the annotation; and a bitmap 40 of the annotation itself. 

At step 41, if the annotation detection is performed locally, then the 
detected objects are transferred from each workstation 14 to the server 12 (or 
if the annotation detection is performed centrally, then the object information 
is already available at the server 12). 

20 At step 42, the objects are communicated from the server 14 to each 

workstation 14, to make all the annotations available to all of the workstations 
for display. 

At step 44, each workstation 14 generates a local display of the 
document with one or more of the annotations overlaid on the document 

25 image, according to display criteria for that workstation 14. The user may 
select viewing of all annotations, or only annotations made by one or more 
certain authors, or selected annotations according to time or date criteria. A 
preferred feature of the present embodiment is the ability to show annotations 
in different colors to distinguish between different authors or originating 

30 workstations. This provides an immediate indication of the author of each 
annotation, and can also enable similar or overlapping annotations to be 
distinguished more easily. 
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Figs. 4(a), (b) and (c) illustrate a document 22 which includes 
handwritten annotations 48a, 48b and 48c made by three different people. 
Fig. 5 shows how the annotations are combined when overlaid together for 
collaborative viewing. By showing the different annotations 48a, 48b, and 48c 
5 in different colors, the annotations are much easier to view and to distinguish 
individually. 

If desired, a user can select a displayed annotation to retrieve associated 
information, for example, the time which the annotation was made (or 
detected), or the name of the person who made it. 

10 At step 46, the system detects any new annotations made at each 

workstation in real time. At each workstation, the document may be imaged 
substantially continuously (or periodically) by means of a camera device. 
Alternatively, the operator at the workstation may decide when to scan a new 
image of a document in order to update any new annotations. 

15 Again, the detection of annotations may be performed locally at each 

workstation 14 or centrally at the server 12 by communicating the captured 
image to the server 12. Real-time annotations may be detected by direct 
comparison of a newly captured image with a previously captured image to 
identify any new markings, which represent the new annotations. Additionally 

20 or alternatively, the algorithm referred to above may be used to identify new 
annotations by discriminating between handwritten markings and machine- 
printed marks. 

Once the new annotations have been identified at step 46, the process 
returns to step 41 at which the new annotations are stored at the server 12 in 
25 the form of new annotation objects. The system then repeats the above- 
described method of distributing new annotation objects to the workstations 
14 for local display in a collaborative display, and the process repeats to 
provide a continuous real-time update. 

The present embodiment can therefore provide a substantially real time 
30 collaborative system, in which there is little, or no, perceptible delay between 
a user marking a document, and having that mark displayed to all users. 



The present embodiment also permits non-handwritten annotations to be 
entered at a workstation, if desired. For example, keyboard entered 
annotations may be entered at a workstation. Additionally, an audio 
annotation (e.g. a voice message) may be recorded, and displayed as an icon 

5 at the desired place in the document. The audio annotation is played 
(reproduced) at any workstation 14 by a user clicking or otherwise selecting 
the icon in his display. 

It will also be appreciated that by identifying and storing the annotations 
as objects, it is possible to "clean" an image of the document, even though the 

10 paper original contains handwritten annotations. In order to clean the image, 
it is necessary simply to remove all of the annotation bitmaps from the image. 
Such a process may be performed either locally at each workstation 14, or 
centrally at the server 12. 

The invention, particularly as illustrated in the preferred embodiment, 

15 provides an extremely powerful and versatile collaborative annotation system, 
which can enable each user to work naturally on a paper copy of the 
document, without requiring each user to enter annotations using a keyboard, 
or using a complicated x-y digitizer pad and digitizing pen. 

It will be appreciated that the foregoing description is merely illustrative 

20 of a preferred non-limiting embodiment of the invention. Those skilled in the 
art will be aware of many modifications and equivalents within the scope and 
spirit of the invention. 
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